Papers with vanilla gradient descent algorithms
Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning (2021.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models can be fine-tuned to produce state-of-the-art results for a wide range of language understanding tasks. |
| Approach: | They propose to analyze fine-tuning through the lens of intrinsic dimension . they show that pre-trained models have a low intrinsic dimension reparameterization . |
| Outcome: | The proposed model can achieve 90% of the full parameter performance levels on MRPC with low data regime. |